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ABSTRACT 



The purpose of this thesis is to analyze the integration of 
Modelling and Simulation (M&S) into the U.S. Army Operational Test 
and Evaluation (OT&E) process. The elements, shortfalls, and 
recurring problems associated with the OT&E system are examined 
with a focus on those that can be addressed by M&S. Current and 
future M&S architectures are outlined to provide a base of 
understanding for the applicability to the OT&E process and issues. 
Analysis of the potential strengths and weaknesses of M&S in 
addressing OT&E problems and issues are presented. Lessons learned 
from past OT&E efforts are also analyzed for process improvement 
through M&S integration. From this analysis, a set of 
recommendations in the area of M&S integration into Army OT&E are 
formulated and offered. 
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I. 



INTRODUCTION 



A. PURPOSE 

The purpose of this thesis is to analyze the integration 
of Modeling and Simulation (M&S) into the U.S. Army 
Operational Test and Evaluation (OT&E) process. The elements, 
shortfalls, and recurring problems associated with the OT&E 



system are 


examined 


with a 


focus 


on those that can 


be 


addressed 


by M&S. 


From 


this 


analysis, a set 


of 



recommendations in the area of M&S integration into Army OT&E 
are formulated and offered. The recommendations address both 
cost effectiveness and adequacy of the integrated M&S approach 
to the OT&E process. 

B . BACKGROUND 

The modern battlefield requires continued advancement in 
virtually all complex weapon systems. Complex systems provide 
the accuracy, reduce the manpower requirement, and create the 
ability to control vast numbers of tasks simultaneously. They 
also produce a challenge to the OT&E process because they are 
extremely difficult to test in an operational environment. 
This difficulty is due to system complexity such as weapons 
tracking systems capable of identifying and tracking multiple 
targets or communications systems capable of simultaneously 
processing thousands of analog and digital messages. This 
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same advancing technology has also accelerated the development 
and useful application of M&S systems. 

Modeling and simulation represents an explosive growth 
industry within the U.S. Department of Defense (DoD). The 
technologies comprising these revolutionary techniques are 
receiving increasingly wider application among the Services, 
and in particular the Army, as the DoD budget continues its 
rapid decline. As the cost of OT&E continues to increase 
with the complexity of the systems evaluated, the application 
of M&S to the OT&E process becomes ever more appealing. 
(Will iams , 1 9 9 3 , p . 1 6 ) 

Counterbalancing the appeal of the relatively low cost M&S 
for OT&E applications are laws, directives, regulations and 
policies mandating live testing. In the test and evaluation 
community, there is a widely held distrust for anything that 
is "simulated" and the term itself brings to mind a 
circumvention of generally accepted testing protocols. 

The connotation of any form of M&S within the test and 
evaluation communities is that shortcomings and failures of 
the evaluated system are being masked by the M&S effort. 
Furthermore, the human element is generally thought to be 
eliminated when M&S techniques are used. The following quote 
from Brigadier General Trifiletti, Commanding General of the 
U.S. Army Test and Experimentation Command (TEXCOM) 
illustrates some of the reasons M&S is not widely accepted: 
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The fog of war cannot be simulated by any computer. The 
effect of blisters on the feet of your soldiers, as well 
as most other human characteristics, are not portrayed in 
a simulation. You've got to get out on the ground with 
soldiers and equipment to understand the capabilities and 
limitations. (Trif iletti, 1994 ) 

• 

C. THESIS OBJECTIVE 

This thesis analyzes Army OT&E with a focus on M&S 
applications. The Army's current OT&E process is analyzed to 
identify persistent OT&E problems and issues, and assess 
current shortcomings that exist in OT&E. Where M&S concepts 
have potential application in enhancing OT&E elements or 
providing solutions for OT&E issues, analysis is offered as to 
the type, extent, and applicability of M&S concepts. 

Analysis of the potential strengths and weaknesses of M&S 
in addressing the elements and issues associated with the OT&E 
process are presented. Lessons learned from past OT&E efforts 
are analyzed for possible process improvement through M&S 
integration. Supporting conclusions and recommendations based 
on the analysis of the thesis are presented. 

D. RESEARCH QUESTIONS 

The primary research question of this thesis is: 

• To what extent can modeling and simulation address the 
recurring problems and issues associated with operational 
test and evaluation? 

The four subsidiary research questions are: 

• Does the operational test and evaluation process lend 
itself to integration with modeling and simulation? 
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• What current and proposed models and simulations are 
candidates for integration with operational test and 
evaluation? 

• Can operational test and evaluation costs be reduced 
through the use of modeling and simulation? 

• Can operational test and evaluation elements be enhanced 
through the use of modeling and simulation? 

E. SCOPE AND LIMITATIONS 

This thesis focuses on M&S applications within the U.S. 
Army OT&E process. The potential application of M&S in this 
thesis is limited to the test and evaluation phase of the 
acquisition process. Specifically, this thesis addresses OT&E 
rather than the developmental test and evaluation (DT&E) 
process . 

This thesis identifies potential opportunities to 
^ integrate U.S. Army operational test and evaluation with 
current and future M&S applications in order to improve the 
Army OT&E process. The laws, regulations, directives, and 
policies pertaining to OT&E are examined to identify where M&S 
integration could be accomplished and under what circumstances 
that M&S is prohibited. Persistent issues surrounding OT&E on 
U.S. Army systems identified by experienced OT&E test 
personnel, are examined with emphasis on applying M&S as 
methods of resolving the issues. 

Current and future M&S architecture is outlined to provide 
a base of understanding for the applicability to the OT&E 
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process and issues. Army wide M&S programs and applications 
across numerous commands are examined and presented. 

Analysis of M&S integration opportunities includes limited 
cost analysis and other value added properties of M&S. 
Networked and distributed systems are emphasized in the M&S 
structure. 

F. RESEARCH LITERATURE AND METHODOLOGY 

Research data were obtained from official Government 
directives and policies, journals, previous theses. United 
States Code, DoD and Army regulations and manuals, and 
personal interviews. Information on current M&S programs was 
obtained from Army Training and Doctrine Command (TRADOC) 
Analysis Centers (TRACs), the Army's Simulation, Training and 
Instrumentation Command (STRICOM), and the Defense Modeling 
and Simulation Office (DMSO). Information on the U.S. Army 
operational test and evaluation process was obtained from the 
Operational Test and Evaluation Command (OPTEC) and the Test 
and Experimentation Command (TEXCOM). Current directives and 
policies guiding the OT&E process were reviewed. 

Research was conducted via personal and telephone 
interviews with cognizant M&S and T&E personnel. Interviews 
with OT&E related individuals centered around the OT&E 
processes and the opportunities for M&S to enhance OT&E 
elements or address OT&E problems and issues. Interviews with 
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M&S related individuals focused on the state-of-the-art in M&S 



and their ability to address the OT&E process. 

G. ACROKYMS 

An extensive listing of acronyms associated with both OT&E 
and M&S subjects is presented in the Appendix. 

H. ORGANIZATION OF THESIS 

Chapter II of this thesis addresses the OT&E process 
including OT&E unique elements. The laws, regulations, 
directives, and policies shaping the OT&E process are also 
presented to illustrate M&S opportunities and limitations in 
Army OT&E . 

Chapter III provides an overview of the Army's M&S thrusts 
and the technologies involved in the M&S efforts. This 
chapter addresses both current and immediate future M&S 
opportunities that may have application in the OT&E arena. 

Chapters IV and V, contain the analysis of M&S 
applications in the OT&E process and the conclusions and 
recommendations offered, respectively. 
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II. THE OT&E PRCXTESS 



A. GENERAL 

The most capable system is useless if it cannot be 
employed by the intended user within doctrine, force 
structure, tactics, and training programs. The OT&E process 
is designed to assess a system's effectiveness in a realistic, 
operational environment. 

The purpose of this chapter is to delineate Army OT&E 
missions, definitions, and organizations. The current role of 
M&S in the OT&E process as well as the laws, directives, 
regulations, and policies guiding the Army OT&E process are 
addressed . 

1. OT&E Missions and Definitions 

The OT&E missions are: To determine the operational 
effectiveness and suitability of a system or to evaluate 
tactical and doctrinal concepts, under realistic combat 
conditions; and to determine if the minimum acceptable 
operational performance requirements, as specified in the 
Operational Requirements Document (ORD), have been satisfied. 
(DoDI 5000.2, 1991, p. 8-5) 

Operational effectiveness and suitability refers to 
the weapon system's combat effectiveness achieved when 
operated and maintained by typical users, within the intended 



7 



doctrine and tactics, and as part of standard organizations 
integrated with other battlefield operating systems. 
Operational effectiveness and suitability are distinct from 
weapon system capabilities (e.g., speed, range, armament 
penetration, etc.) determined during DT&E. 

Realistic combat conditions are those that are 
representative of the environment, doctrine, level of 
training, and structure in which the evaluated system would 
normally be expected to operate within a combat environment. 
Realism is one of the critical elements of the OT&E process 
and is discussed in detail later in this chapter. 

2. OT&E Organization 

OT&E is organized as depicted in Figure 1. Congress 
passed legislation creating the Office of Operational Test and 
Evaluation in 1983 with the mission to evaluate the Services' 
weapon systems tests and assess test results. As part of 
that legislation, the Director of Operational Test and 
Evaluation (DOT&E) reports directly to the Secretary of 
Defense and is given broad authority to suspend major weapons 
programs that perform poorly in operational tests. In 
addition to reporting to the Secretary of Defense, the DOT&E 
reports directly to both the House and the Senate Armed 
Services Committees, at least annually, regarding OT&E of 
weapon systems being developed. The goal of the legislation 



8 




BROKEN UNE INOICATE3 RE PORTING CHANNEL FROM OOTE TO CONORESS 



Figure 1 OT&E Organization 



is to save time and money by exposing problems before 
expensive weapons are purchased and fielded. 

The Operational Test and Evaluation Command (OPTEC) 
was established 15 November 1990 by Secretary of the Army 
General Orders Number 6. It consists of the OPTEC 
Headquarters and Support Agencies, the Operational Evaluation 
Command (OEC) and the Test and Experimentation Command 
(TEXCOM). The new command consolidates previously designated 
commands and agencies including the TRADOC Test and 
Experimentation Command (TEXCOM), the former Operational Test 
and Evaluation Agency (OTEA), and the former Acquisition and 
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Development of Threat Simulators Activity (ADATS-A) into a 
single command. ADATS-A, renamed the Operational Threat 
Support Activity (OTSA), and the Test and Evaluation 
Coordination Offices (TECOs), are incorporated within OPTEC 
Headquarters. OPTEC 's mission is to conduct and monitor user 
test and evaluation (except medical) for the Army. User T&E 
includes initial and follow-on operational test and 
evaluation (lOTE and FOTE) in support of the materiel 
acquisition process, force development testing and 
experimentation (FDTE), concept evaluation program (CEP) 
trials, early user test and evaluation (EUTE), and the Army 
part of joint test and evaluation (JT&E). 

OPTEC, Figure 2, is a field operating agency of the 
Office of the Chief of Staff of the Army. In keeping with the 
Defense Directives, OPTEC reports the results of Army OT&E 
through the Vice Chief of Staff of the Army directly to the 
Army and Defense leadership. The main part of OPTEC 's mission 
is the planning, conducting, and reporting of Army OT&E which 
has been required by law since 1972. Additionally, OPTEC 
conducts tests for TRADOC in support of its mission to develop 
combat doctrine, organizations, and materiel requirements. 

3. OT&E Support to the Acquisition Process 

The materiel acquisition process can take many years 
from the time a materiel requirement is identified until the 
system is fielded. Although in this process, OT&E accounts 
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OPTEC 




Figure 2 OPTEC Organizational Structure 

for only a short time period/ the results weigh heavily on the 
decisions to continue development, accept the system, or 
change organization, doctrine, and concepts. 

The fundamental purpose of OT&E in the acquisition 
process is to identify the areas of risk associated with user 
requirements and acceptance to be reduced or eliminated. 
During the early phases of development, T&E is conducted to 
demonstrate the feasibility of conceptual approaches, minimize 
design risk, identify design alternatives, and estimate 
operational effectiveness and suitability. As a weapon system 
progresses through the developmental process, T&E emphasis 
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turns from DT&E towards OT&E, even though DT&E and OT&E may 
not be conducted sequentially. (DSMC TEMG, 1988 , p . 1-1 ) 

After a weapon system has successfully completed its 
development phase, there is tremendous political and 
bureaucratic pressure to begin full-scale production. The 
operational testers are often viewed as "show stoppers" 
because successful completion of OT&E is required before any 
developmental system can progress beyond low rate initial 
production (LRIP). The problem is compounded by the current 
acquisition process which schedules critical OT&E events just 
before the production decision milestone. 
(LeSueur, 19 94, p. 12) 

Figure 3 depicts Army OT&E events in each acquisition 
phase. The lOTE block includes CEP, EUTE, and FDTE events. 



OT&E IN THE ACQUISITION PROCESS 
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Figure 3 OT&E Support to the Acquisition Process 
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During the Concept Exploration Phase prior to 
Milestone I, laboratory testing, modeling and simulations are 
conducted to demonstrate and assess the capabilities of key 
subsystems and components. Studies, analyses, simulation, and 
test data are used to explore and evaluate alternative concept 
designs proposed to satisfy user generated requirements. OT&E 
conducted during this phase, called early operational 
assessment, investigates deficiencies identified during the 
mission area analysis. OPTEC monitors concept exploration T&E 
for future T&E planning and to provide effectiveness and 
suitability inputs desired by the Program Manager. 
Operational assessments addressing the operational impact of 
candidate technical approaches are conducted. 

During the Concept Demonstration/Validation phase, 
operational effectiveness and operational suitability 
assessments are conducted. Information on tactics, doctrine, 
personnel requirements, and organization impacts of the weapon 
system are gathered. OT&E assessments are used to support the 
Milestone II decisions for developing promising alternatives. 

The objective of the Engineering and Manufacturing 
Development phase is to design, fabricate and test systems 
that closely approximate the final product. Prior to the 
Milestone III decision, a dedicated OT&E is conducted on 
equipment formally certified as ready for "final OT&E". OT&E 
has the greatest impact on major programs in this phase 
because the decision to proceed beyond LRIP is contingent on 
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successfully completing the lOTE. A formal Operational 
Evaluation (OPEVAL) is required for Milestone III. The 
purpose of OT&E during this phase is to: 

• Estimate the operational effectiveness and suitability of 
the system. 

•Identify operational deficiencies. 

• Recommend and evaluate changes in production 
configuration . 

• Provide information for developing and refining logistics 
support requirements. 

• Estimate the survivability of the system in the 
operational environment. 

Post-production OT&E are Follow-on Operational Test 
and Evaluation (FOTE) programs designed to verify the 
operational effectiveness and suitability of the production 
system and to determine if deficiencies identified during lOTE 
have been corrected. FOTEs also refine doctrine, tactics, 
techniques, and training programs for the life of the system. 
(DSMC TEMG,1988) 

B. OT&E LAWS, REGULATIONS, AND DIRECTIVES 

The OT&E process is guided by numerous laws, regulations, 
directives, and policies. The provisions of these mandates 
and guiding vehicles directly affect how the OT&E system 
plans, conducts, and analyzes evaluations. The role of M&S in 
the OT&E process is limited and, in some cases, precluded by 
the mandates guiding OT&E. 
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The current downsizing of the Army creates pressure to 
reduce costs at every level, including within the OT&E 
process. Even with these economic pressures, some lawmakers 
are continuing to guard against reductions in OT&E testing. 
In his article Fallout: Weapons that don't work?, Rick Maze 
reported the following: 

Senators David Pryor (D-Ark) and William Roth (R-Del) said 
they oppose a Pentagon procurement reform plan that would 
allow operational tests to be waived if they are too 
expensive, cause long delays or otherwise interfere with 
the purchase of systems. 

Pryor and Roth were the chief sponsors of the 1983 law 
that created the Pentagon's Office of Testing and 
Evaluation and remain committed to the principle that 
weapons need to be fully tested in live-fire exercises 
before large numbers are built. "It is a very important 
part of our military preparedness to make sure these 
weapons work", said Pryor. (Maze, 1994 , p. 20 ) 

Law makers and policy makers have reinforced both the 
requirement for OT&E and the guiding documents addressed in 
the following paragraphs. 

1. Title 10, United States Code 

United States Code, Title 10, Chapter 138 has two 
sections that pertain to the OT&E process. Section 2399, 
Operational test and evaluation of defense acquisition 
programs , and section 2366, Major systems and munitions 
programs: survivability and lethality testing; operational 

testing. 



Section 2399 establishes the conditions necessary for 
a major program (acquisition category I and II) to proceed 
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beyond LRIP and the DOTE reporting responsibilities. This 
law's provisions state that a major defense acquisition 
program may not proceed beyond low rate initial production 
(LRIP) until the Initial Operational Test and Evaluation 
(lOTE) for that program is complete. It further directs that 
the DOTE shall analyze the results of major systems' OT&E and 
prepare a report for the House and Senate Armed Services 
Committees and the Secretary of Defense. The report shall 
state the DOTE's opinion regarding OT&E adequacy and whether 
the results indicate that the evaluated system is effective 
and suitable for combat. (USC, 1989 ,pp. 230-231 ) 

Section 2366 directs the Secretary of Defense to 
ensure that no major system proceed beyond LRIP until the lOTE 
and survivability/lethality testing is complete. This section 
specifically defines the conditions required for survivability 
and lethality testing. These conditions include the 
requirement for live firing on representative systems and 
therefore, preclude any major system from being acquired 
solely on the basis of M&S. Contractor and other personnel 
not normally involved in the operation or support of a weapon 
system, are forbidden to participate in OT&E under the 
provisions of this section. (USC, 1989, pp. 581-582) 

2. Department of Defense Instruction (DoDI) 5000.2 

This instruction references and reinforces the 
provisions of Title 10, United States Code, outlined in 
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paragraph one, above. It directs that OT&E shall be designed 
to support the decision to proceed beyond LRIP and establishes 
four objectives for OT&E testing: 

• Provide essential information for assessment of 

acquisition risk and for decisionmaking. 

• Verify attainment of technical performance specifications 
and objectives. 

• Verify that systems are operationally effective and 
suitable for intended use. 

• Provide essential information in support of 

decisionmaking. 

With regard to the use of M&S in the OT&E process, 
DoDI 5000.2 interprets Title 10 in a strict manner. The 

Instruction states that OT&E does not include an operational 
assessment based exclusively on computer modeling, simulation, 
or analyses of program documents. (DoDI 5000 . 2 , 1991 , p. 8-2 ) 
3. Army Regulation 73-1, Test and Evaluation Policy 

Army Regulation 73-1 (AR 73-1) directs the 

implementation of DoD Directive 5000.1, DoD Instruction 
5000.2, and DoD Manual 5000. 2-M. This regulation describes 
the type of OT&E applications to be provided in support of 
each phase and milestone. It links T&E to the acquisition 
process and directs T&E design to support the acquisition 
phases and milestones. OPTEC is charged with the overall 
management of Army OT&E programs through use of an operational 
tester (TEXCOM) and an operational evaluator (OEC). 
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C. CURRENT ARMY OT&E AND M&S INTEGRATION 



Through its stated policies, OPTEC recognizes the use of 
M&S to achieve adeguate realism, support economical test 
execution, and provide for sufficiently adequate evaluations. 
However, to ensure that OPTEC maintains its objective, 
independent evaluator perspective, every M&S application used 
in the OT&E process must be subjected to Verification, 
Validation, and Accreditation (W&A) before any data or 
information can be used. M&S applications must be accredited 
for each specific application, and the W&A conducted for one 
OT&E does not apply to another. Because of the W&A 
requirement, there is no listing or catalog of M&S 
applications available for use in the OT&E process. (OPTEC 
73-21-1,1993) 

D. OT&E ELEMENTS 

The OT&E process is guided by operationally related test 
and evaluation elements. These elements are unique to the 
OT&E process and drive the design of all OT&E programs. 

1. Realism 

Realistic environments are essential to achieving the 
goals of OT&E. A realistic environment is one that is 
representative of the conditions, doctrine, level of training, 
and structure in which the evaluated system would normally be 
expected to operate, in a combat environment. These 
environments are distinctly different than those encountered 
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in the DT&E phase where the environment is carefully 
controlled. 

Realism in the OT&E test process is affected by the 
resources available to replicate the representative threat and 
friendly force array, create the desired combat battlefield 
environment, and stress the evaluated system over time. Other 
factors affecting realism include the degree to which test 
participants represent typical operational personnel, test 
personnel familiarity with test ranges and maneuver areas, and 
limitations created by safety or environmental concerns. 

The conditions present during the OT&E must stress the 
evaluated system within the doctrinal envelope of operation. 
"Stimulators" are used to stress communications and software 
intensive components of the evaluated system to levels 
expected in a combat environment. 

An OT&E of a system includes its interoperability 
characteristics. A realistic structure of associated 
battlefield operating systems is essential. 

2. User Oriented 

An integral part of the OT&E process is the user. 
Representative personnel having the correct organizational 
grade and specialty structure, level of training, experience 
and aptitudes are required for an unbiased operational 
evaluation of the system. 
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Users include personnel that operate, maintain, 
support, or provide command and control functions affecting 
the evaluated system. This requires that the interrelated 
battlefield operating systems and the evaluated system's 
support structure personnel must also be representative. 

The term "users" specifically excludes contractor and 
other personnel who would not normally be involved in the 
operation, control, or maintenance of the evaluated system. 
To avoid bias, it is important that the personnel selected to 
represent the user are indeed representative. 

3. Representative Systems 

The evaluated system equipment must be sufficiently 
mature to be considered functionally representative of the 
versions eventually fielded. Software must be complete and as 
near to the fielded version as possible when the OT&E is 
conducted. All associated equipment, publications, training 
programs, and test/measurement/diagnostic equipment (TMDE) 
should be complete prior to the evaluation. 

The evaluated system is usually part of a larger table 
of organization and equipment (TO&E) and interacts or is 
supported by other standard systems. These systems must also 
be representative of those found in an operational 
environment . 
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4. Sufficiency 

Sufficiency in the OT&E process ensures that the test 
plan addresses all of the issues specified in the Operational 
Requirements Document (ORD), the data collection plan provides 
for sufficient data, and the evaluation plan is sufficient for 
the system evaluation. The structure of the OT&E vehicles for 
acquiring and reducing the information is critical for a 
complete and unbiased evaluation. 

The test plan is the key document in determining what 
data, and under what circumstances data are collected. The 
test plan is driven by the user provided issues specified in 
the ORD and the requirements for data supporting the 
evaluation. Sufficient, unbiased iterations of test events 
must be accumulated to facilitate the evaluation techniques 
prescribed in the evaluation plan. 

E. SUMMARY 

The OT&E process is designed to assess the effectiveness 
and suitability of systems, concepts, doctrine, and tactics in 
a realistic, combat environment. The results of OT&E 
assessments have significant impact on the acquisition of 
major programs and have visibility at the Congressional level 
as well as the top echelons of the Army. 

The use of M&S in the OT&E process is limited by law. DoD 
and Army policies, designed to protect the objectivity of 
OT&E, reflect a bias against the use of M&S. Counterbalancing 
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that bias are budgetary constraints that negatively impact on 
the scope of "live" testing. 

Advancing technologies have vastly improved the 
capabilities of modern M&S applications. The next chapter 
addresses the Army M&S thrusts that are attempting to exploit 
these advancing technologies. 
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III. ARMY MODELING AND SIMULATION THRUSTS 



A. GENERAL 

From the Army's point of view, anything short of actual 
combat is simulation. Modeling and Simulation (M&S) 
encompasses applications ranging from simple mathematical or 
physical models to sophisticated systems integrating three 
simulation types; "live", "constructive", and "virtual". The 
three types of simulations are defined below: 

1. "live" simulation consists of operations with users 
employing real equipment in the field 

2. "constructive" simulation which deals with wargames, 
models and analytical tools 

3. "virtual" simulation refers to systems and troops in 
simulators on synthetic battlefields ( Singley, 1993,p. 35 ) . 

This chapter addresses M&S concepts and applications that 
are currently available or are being planned for use by the 
Army in the immediate future. The M&S applications described 
are not the only resources available, but are representative 
of the technology available and are therefore used to 
illustrate applications. 

The Army's M&S resources have been primarily used in the 
areas of training, testing, material development, combat 
development, and analysis. The Army has increased the 
capability of its high resolution modeling facilities and is 
making progress in the area of three dimensional simulation. 
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These advancements have generated high expectations that, 
within the next few years, the M&S community will field the 
reguisite fidelity and distributed capabilities needed to 
streamline the current acquisition process. (Crouch, 1994, p. 3 ) 
Technological components of the Army simulation capability 
include a Distributed Interactive Simulation (DIS) environment 
transmitted over the Defense Simulation Internet 
communications network. DIS provides protocols that enable 
the communication between various models such as Janus (a high 
resolution constructive model), semi-automated forces (SAFOR) 
generator, and the Battlefield Distributed Simulation - 
Developmental (BDS-D) virtual simulator complex. 

The highest echelons of Army leadership recognize the need 
to exploit advanced M&S capabilities for weapon system 
acquisition in the post-cold war era. In the "United States 
Army Posture Statement FY95” , The Honorable Togo D. West, Jr., 
Secretary of the Army, and Army Chief of Staff General Gordon 
R. Sullivan state: 

The Army will maintain technological superiority through 
pursuit of promising advanced technologies and concepts, 
developing new systems when existing systems have reached 
the end of their useful lives or when a new system offers 
an essential, revolutionary combat capability. We will 
exploit Advanced Distributed Simulation for better, more 
affordable requirements and acquisition testing and will 
reduce acquisition costs by reducing infrastructure and 
development cycle times. (West, 1994, p. 87 ) 

They also outline two major assets the Army will use to 

develop weapon systems; the Louisiana Maneuvers (LAM) and the 

Battle Labs. Both of these assets integrate M&S into the 
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materiel and concept development process and will be discussed 
in detail later in this chapter. 



B. DoD M&S POLICIES 

The Army is adopting policies to take advantage of 
advanced M&S opportunities. The Army's leaders are setting 
the M&S policy cornerstones as evidenced by Army Chief of 
Staff, General Gordon Sullivan's statement: 



You need to know that we will use simulation techniques 
throughout the Army's acquisition process. We will 
determine needs in large-scale, simulation-supported 
exercises that allow us to consider alternative solutions 
that meet our needs. We will use drawings, diagrams and 
3-dimensional models generated by computers, put them in 
constructive or virtual environments, and compare 
alternatives both technically and tactically. The most 
promising technologies will be tested by real soldiers, 
first in reconf igurable crew stations, then in full scale 
simulations. Final designs, production and assembly steps 
will also be simulated in virtual factories before actual 
prototypes are made. Then the actual and virtual 
prototypes will be exercised simultaneously to discover 
potential problems before production begins. Tactics, 
techniques, and procedures are also developed along with 
the system so that the system is fully ready for use when 
produced. (Sullivan, 1993 ) 

Lieutenant General Willieim H. Forster, the Military Deputy 
to the Assistant Secretary of the Army (Research, Development 
and Acquisition), further refined the M&S focus in his 24 May 
1993 Memorandum for the Deputy Commanding General, U.S. Army 
Materiel Command and all Program Executive Officers. Under 
the subject of "Simulation Support to the Army", LTG Forster 
stated: 
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The Army Science Board and Defense Science Board have 
recently studied the potential improvements to DoD 
acquisition offered by advanced simulation, particularly 
Distributed Interactive Simulation (DIS). Both concluded 
that simulation can improve acquisition from concept to 
fielding through such innovations as: virtual 
prototyping; engineering simulation; linking of 
constructive, virtual and/or live simulations; assisting 
the user in execution of experiments in employment 
tactics; user test design and critical issue 
identification; and improved training prior to fielding. 

The Army is leading the way for DoD in simulation with 
such initiatives as Battlefield Distributed Simulation- 
Developmental, Close Combat Tactical Trainer, and DIS 
Modernization and Master Plans. We need to take full 
advantage from concept to fielding. Effective second 
quarter fiscal year 1994, all Army acquisition strategies 
for Acquisition Category I and II programs will include a 
simulation support plan. Additionally, the simulation 
support plan must be included in the Program Manager's 
ASARC briefing. Other programs may be tasked by the Army 
Acquisition Executive to include a simulation support 
plan. (Forster, 1993, p. 1 ) 

Policies integrating M&S into the acquisition process have 
been initiated from the top levels of the Army. The Battle 
Labs and LAM have been established and charged with pursuing 
new technologies and investigating concepts using an 
integrated M&S approach. Policies for integrating M&S into 
the OT&E portion of the acquisition process have begun, but 
not fulfilled. 



C. M&S CONCEPTS AND APPLICATIONS 

The Army plans to link M&S technological components 
together through networks such as the Defense Systems Internet 
(DSI) to create environments like the Distributed Interactive 
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Simulation (DIS) system. These M&S component concepts and 
assets are discussed in detail in the following sections. 

1. Model-Test-Model 

Model-Test-Model (MTM) is a three phase process 
designed to assist DoD and the Army in conducting more 
effective and efficient test and evaluation of new weapon 
systems. The concept envisions the synergistic interaction of 
testing and modeling to enhance both operational testing and 
combined arms modeling. The first phase examines the test 
design to identify possible efficiencies and effectiveness 
improvements of the design. The second phase examines the 
conduct of the test to gain insights into potential model-test 
differences. The final phase focuses on the amount of 

correlation or association between the model and test results. 
Based on the amount of correlation, the model may be 
accredited for a specific trial. Once trials or missions are 
accredited, test results may be extended beyond test scenarios 
and conditions which may be cost prohibitive or constrained 
for environmental or safety reasons. (IR MTM, 1994) 

A basic building block of MTM is a model, which is 
defined as: 

A representation of an actual or conceptual system that 
involves mathematics, logical expressions, or computer 
simulations that can be used to predict how the system 
might perform or survive under various conditions or in a 
range of hostile environments. (DSMC TEMG, 1988,p.B-8) 
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2. Janus Combat Model 



Janus is a computer-based, two-sided combat simulation 
model. Janus UNIX 3.17, and Janus Virtual Memory System (VMS) 
4.0, are the most recent versions of the model that are now in 
common use. (Pate, 1992 ,p. 2 ) 

The Janus combat model accommodates up to 600 
individual combat systems, including up to 100 indirect fire 
systems, for both threat and friendly forces. All systems are 
capable of moving, detecting, and firing over a 50 square 
kilometer, three dimensional terrain representation. Combat 
systems ranging from major individual fighting platforms such 
as tanks and helicopters to dismounted infantry, are portrayed 
using the attributes of the real or notional systems being 
modeled (e.g., size, speed, sensors, armament, ballistic 
protection, thermal/optical contrast, etc.). The 
vulnerability of each system is characterized by data sets of 
probability of hit (Ph) and probability of kill (Pr) that 
individually associate each combat system with each weapon in 
the simulation. (Crooks, 1992 ) 

Janus, because of its high resolution capabilities, 
has been primarily used in aiding analysts with Cost and 
Operational Effectiveness Analysis (COEA) and combat 
development system studies such as the 9th Infantry Division 
(Motorized) force design. Recently, the concept of 
integrating a constructive model (Janus) with a virtual 
simulation environment was explored. One of the Anti-Armor 
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Advanced Technology Demonstration thrusts is to merge 
constructive and virtual simulation worlds. This allows the 
orchestration of selected forces, within the virtual 
simulation environment, to be controlled from a constructive 
model such as Janus. This orchestration is referred to as 
semi-automated forces. (Crouch, 1994, pp. 49-50) 

3 . Semi-Automated Forces 

The semi-automated forces (SAFOR) capability allows a 
single individual operating a constructive model to control 
various sized units such as platoons, companies, or battalions 
within a virtual simulation. These forces appear on the 
virtual battlefield just as manned simulators do; the fact 
that they are SAFOR is transparent to the other participants. 
To keep costs associated with experiments within budgetary 
constraints, SAFOR can be used to represent both friendly 
(adjacent, supporting, higher echelon and lower echelon 
elements) and threat forces. (Loral, 1992, p. 2 ) 

SAFOR is useful from both a command and control and a 
cost savings perspective. Command and control is enhanced by 
the capability to control several systems on the virtual 
battlefield from a constructive model. Savings are derived 
from constructive models that generate a system on the virtual 
battlefield at a lower cost than those generated by virtual 
networks such as BDS-D. (Crouch, 1994 ,p. 51 ) 



29 



4. Battlefield Distributed Simulation - Developmental 



Battlefield Distributed Simulation - Developmental 
(BDS-D) provides the technological framework to conduct a 
simulated battlefield over a distributed network. Through 
this system, individuals can fight and analyze the 
effectiveness differences resulting from the changes made in 
equipment, doctrine, tactics, organizations, and training 
methods. The sequence of the battle can be recorded and later 
analyzed in detail to refine those changes. 

The supporting technologies create a simulated or 
“virtual" battlefield on which users can conduct cost 
effective experiments or training exercises. The exercises 
are conducted using actual soldiers operating simulators, 
permitting soldier-in-the-loop experimentation. Through a 
combination of local area and extended distance networks, 
soldiers operating simulators at one site are able to see and 
interact with soldiers operating at other sites on a common 
digitized battlefield. (Loral , 1992 ,p. 3 ) 

The BDS-D program is sponsored by the U.S. Army and 
the Commander, U.S. Army Simulation, Training, and 
Instrumentation Command (STRICOM) is the Program Manager for 
the BDS-D effort. STRICOM provides the focal point between 
DoD agencies, user agencies, industry, and the BDS-D sites. 
(Loral , 1992 , p . 4 ) 

BDS-D will support experiments and evaluations in a 
variety of areas. Using the approach of simulating before and 
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during procurement of a new- weapon system, users are able to 
experiment with the weapon system design throughout its 
acquisition life cycle. For example, developers can perform 
the following: 

•Define requirements accurately and assess trade-offs. 

• Explore the capabilities that should be incorporated into 
a new or existing system. 

• Investigate the density and allocation of the system that 
achieves optimum performance on the battlefield. 

• Determine the best means to employ the system once it is 
built (Loral , 1992 , p. 4 ) . 

Users can experiment with new and innovative ways of 
employing weapon systems so that they better realize their 
design potentials. Changes in organizational structure can 
also be analyzed to determine the relative effectiveness on 
the battlefield of competing organizations. ( Loral , 1992 ,p. 4 ) 
Local area networks consisting of low cost battlefield 
simulators, and simulations of experimental systems, and SAFOR 
are required to facilitate this effort. These simulators will 
be linked together through a network such as DSI to provide 
virtual combat operations in the DIS environment for materiel 
and combat development, and operational testing exercises. 
(Kelly, 1993, p. 19 ) 

5. Distributed Interactive Simulation 

One of the challenges is integrating constructive and 
virtual simulation technologies. The dissimilar computer 
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systems and simulators that the Army intends to combine into 
one integrated network were not designed to communicate with 
one another. To achieve this desired seamless simulation 
capability, a simulated environment that provides standard 
terms and protocols which allows these different computer 
systems and simulators to communicate is required. The 
standardization of terms is achieved by Distributed 
Interactive Simulation (DIS) Protocol Data Units (PDUs). 
(Crouch, 1994,p.56) 

DIS creates a synthetic environment within which 
humans may interact through simulation at multiple, networked 
sites using compliant architecture, modeling, protocols, 
standards, and data bases. DIS and its PDUs are the next 
generation of distributed simulation evolving from the 
Advanced Research Project Agency's (ARPA) research project of 
the 1980 's known as Simulation Network (SIMNET). The DIS is 
in its initial stages with many obstacles yet to overcome. 
(1ST, 1993, p. 4) 

DIS will take advantage of currently installed and 
future simulations manufactured by different organizations. 
Consequently, interoperability between dissimilar simulations 
is absolutely required. The first step in achieving this 
interoperability is to develop a communications protocol. 
There must be a standard set of messages that communicate 
between host computers, the states of simulated and real 
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entities, and their interactions. This information is 
communicated through DIS PDUs. ( 1ST, 1993,p. 8 ) 

6. Defense Simulation Internet 

DIS operations are supported by a communication system 
known as the Defense Simulation Internet (DSI). This 
communication system was developed, and is currently operated 
by Advanced Research Projects Agency. The DSI consists of 
commercial telephone circuits over existing networks with 
nodes at user locations, and strategically placed switching 
nodes (Fix East and Fix West) with a central controlling 
facility in Chicago, Illinois. Connectivity is made with 
military and civilian satellites to allow worldwide, 
simultaneous DIS operations. (DIS MODPLAN, 1993 , p. 2 ) 

There are approximately 30 DSI nodes supporting all 
Services' command posts. Battle Simulation Centers, test beds. 
Battle Labs, research centers, unified commands, and civilian 
contractors that support the military. The DSI is expected to 
expand over the next year with approximately 25 additional 
sites. (DIS MODPLAN, 1993, p. 2) 

Each location connected to the DSI network is referred 
to as a DSI node. The Army operates two TRADOC Battle Lab 
nodes on the DSI; Fort Knox and Fort Rucker with Fort Rucker 
designated as a Battle Lab support facility. Eventually the 
Army wishes to operate six to eight Army Battle Lab nodes and 
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additional communications nodes at most major commands 
throughout the Army. ( Singley , 1993 ,p. 37 ) 

Through the interpretive capabilities of DIS and the 
communication network established by the DSI, the Army will 
attempt to link constructive models such as Janus with virtual 
simulation networks such as BDS-D. (Crouch , 1994 ,p . 57 ) 

D. BATTLE LABS 

TRADOC has organized six Battle Labs to identify, develop, 
and experiment with new warfighting concepts and new 
capabilities offered by emerging technologies. The Battle 
labs are distributed as depicted in Figure 4. 




34 



The Battle Labs initiative is a response to the 
unpredictability of the post-cold war world. The wide variety 
and rapidly changing array of direct and indirect threat 
situations have replaced the single, well-defined threat of 
the Warsaw Pact which drove doctrine and materiel requirements 
during the cold-war era. (BLG, 1993,p. 3 ) 

Battle Labs maintain the Army perspective across a wide 
spectrum of ideas and concepts to ensure that the Army remains 
dominant on future battlefields. Through conceptualizing, 
analyzing, simulating, testing, and evaluating projects. 
Battle Labs aggressively seek out emerging concepts and 
technologies worldwide. They streamline the Army planning 
process by providing an organized way to define requirements; 
allowing industry to develop a focus for developmental work; 
providing industry access to a pool of Army thinkers who can 
delineate ideas about modernization alternatives; and linking 
combat lessons learned, exercise results, and insights from 
the regional Commanders in Chief to a formal analytical 
testbed. (West, 1994 ,p. 78 ) 

Battle Labs will be linked through the DSI to each other, 
the Army R&D Community, sister Services, DoD, and national 
agencies. All are organizing to take advantage of the DIS 
technology, which was not available to earlier test-bed 
operations. BDS-D networks will allow warfighting experts at 
TRADOC Centers to advance ideas and test them by simulation at 
a number of locations. (BLG, 1993, p. 4) 
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Battle Labs will provide the tools and standards to 
simulate activities, at a high level of realism, from theaters 
of war to factories and manufacturing processes. The 
mechanism for entry into the synthetic battlefields has been 
limited to a few networked simulators and individual 
workstations. These will be greatly expanded in the 1990s to 
include the reconf igurable BDS-D simulators that will provide 
the desired mix of real ranges, virtual simulations and 
aggregated constructive simulations into wargame 
representations. Multipurpose surrogates, such as SAFOR, 
supported by computer emulation, will allow soldiers to 
participate or to be simulated in battles. (BLG, 1993,p. 6) 

Simulation tools and methodologies integrated into Battle 
Labs offer industry a new area of innovative development that 
has potential beyond its military application. The use of 
concurrent engineering principles reduces development time 
and speeds the acquisition process. Virtual prototypes will 
be produced, so design and manufacturing tradeoffs can be 
evaluated. Eventually the manufacturing process, the military 
system, and the system's performance may all be modeled and 
refined before the first piece of hardware is built. 
(BLG, 1993, p. 6) 

Synthetic environments will not completely replace 
hardware demonstrations as a means of introducing new 
capabilities to the user. However, given the increased costs 
of hardware development and test, contrasted with the 
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decreased costs and increasing fidelity of reconf igurable 
simulators, synthetic environments become ever more appealing. 
(BLG, 1993, p. 7) 

Battle Labs will help prepare the Army for the challenges 
of the next century. Unlike the manpower-intensive Louisiana 
Maneuvers of the 1940s, simulation and Battle Labs will afford 
a basis for the Louisiana Maneuvers of the 1990s and beyond. 
(Ross, 1993, p. 18) 

E. LOUISIANA MANEUVERS 

The term Louisiana Maneuvers (LAM) refers to a series of 
large-scale military exercises carried out by Army General 
George C. Marshall in Louisiana in 1941 to rebuild and renew 
the Army for combat in World War II. The maneuvers met the 
challenge by training soldiers, units, and headquarters; 
validating and improving doctrine and equipment; and testing 
new units and concepts. 

The Army reestablished the LAM in 1992 under a Chief of 
Staff of the Army charter: 

The Louisiana Maneuvers will energize and focus the Army 
on warfighting and its Title X responsibilities, will 
provide the Army's senior leadership strategic agility in 
decision making, and will assess the Army's direction and 
progress on its journey into the 21st Century. 

LAM allows the Army to think, to grow, and to effectively 
take charge of the process of change. It provides the 
Army leadership a systematic approach to explore and 
examine doctrine, organization, training, materiel, leader 
development, and soldier issues shaping the force for the 
next century, without putting thousands of soldiers in the 
field. By harnessing the power of the microprocessor, the 
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Army will use simulations to develop and produce new 
equipment, to enhance combat readiness, to train, and to 
experiment with new ideas. LAM uses exercises, battle 
laboratories, and other mechanisms to cover the full range 
of military operations. (West , 1994 , pp. 76-77 ) 

Some major issues that LAM will exeimine in the near future 
include an effectiveness review of command, control, 
communications, computers, and intelligence (C4I) 
architectures; implications of around-the-clock operations; 
the impact of new technologies and equipment on the 
battlefield; deployability, lethality, and survivability of 
light and heavy forces; the impact of weapons of mass 
destruction in today's security environment; and a broad range 
of force generation and deployment requirements. 
(West, 1994, p. 77) 

Relying heavily upon simulation and modeling technologies, 
senior Army leaders will use LAM as a tool to help save money 
by speeding the introduction of promising new weapon systems. 
This is derived from quickly eliminating unworkable concepts, 
aiding in the development of new doctrine, and generally 
guiding the Army as it reshapes itself for meeting post-cold 
war missions. (Holzer, 1993,p. 36) 

F. SUMMARY 

The Army is moving to take advantage of advanced M&S 
opportunities and the potential cost savings M&S offers in a 
fiscally constrained post-cold war era. The Army's leadership 
is setting policies that endorse M&S proliferation Army wide, 
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including in the costly acquisition process. As a vital part 
of the acquisition process, how can OT&E integrate M&S 
concepts and applications to address recurring problems and 
shortcomings, and enhance the OT&E process? 
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IV. ANALYSIS 



A.* GENERAL 

The OT&E cominunity has a most difficult role in defining 
when, where and how it can participate outside the live 
environment. Operational testers find themselves caught 
between United States Code, Title 10 mandates and the spiral 
of diminishing defense dollars. This is a most dangerous 
position. (Crouch, 1994, p. 86) 

The "most dangerous position" refers to a situation where 
the OT&E independent evaluator (OPTEC) does not integrate M&S 
and continues to use expensive live testing, while materiel 
and combat developers push for OT&E certification based on 
their own, cost effective operational M&S applications. 
Although undesirable, the loss of the independent OT&E view 
could result from the continuing economic pressures in the 
post-cold war era. 

The Defense Modeling and Simulation Office (DMSO) is 
funding simulation efforts to test the merits of using other- 
than-live environments for operational testing. The customers 
for this effort are primarily the materiel and combat 
development communities. Data are still insufficient to 
assess the current ability to utilize other-than-live 
environments for operational testing. However, technology 
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advancements in the immediate future are expected to provide 
the required simulation fidelity that will make virtual 
simulation a viable alternative for some early and follow-on 
portions of operational assessments. (Crouch, 1994, p. 87) 

This chapter examines OT&E issues and problems, and 
analyzes M&S capabilities to resolve them or reduce their 
severity. The methodology used and OT&E problem and issue 
analysis are addressed in the following sections. 

B. METHODOLOGY 

The major OT&E problems and issues were established by 
conducting interviews with experienced OT&E professionals 
assigned to the TEXCOM Experimentation Center (TEC) located at 
Fort Hunter-Liggett , California. The OT&E problems and issues 
were identified from OT&Es planned or conducted at TEC, Fort 
Hunter-Liggett . 

1. OT&E Weapon Systems 

OT&E tests were planned and conducted on the six 
weapon systems briefly described below: 

• M1A2 Abrams Main Battle Tank. Upgrades to the Ml Abrams 
include improved computer and armor protection systems . 
This weapon system is designed to be fielded in armored, 
mechanized infantry, and cavalry units, ranging from 
platoon to division sized elements. 

• Air-to-Air Target Designator. This device uses laser 
technology to designate airborne targets for armed scout 
and attack helicopters assigned to air attack and air 
cavalry units. 

•Multi-Spectral Combat Decoy. This system provides visual 
and thermal tracked vehicle signatures in a lightweight 
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kit form for use by armored, mechanized infantry, and 
counter-intelligence units. 

•Mini Eyesafe Laser Infrared Observation System (MELIOS). 
MELIOS was designed to improve the ni^t observation 
capabilities for combat, combat support, and combat 
* service support units at every echelon from platoon 
through corps level. 

• Javelin Anti-Tank Missile System. The Javelin is 
replacing the shoulder-fired Dragon missile system and 
provides extended range and improved attack attitudes. It 
will replace all Dragon systems in combat and combat 
support units from platoon through corps level. 

• AH-64D Longbow Apache Helicopter. Upgrades to the AH-64D 
Apache include improved armament (including Longbow 
Hellfire missiles), automated systems, engines, and fire 
control radar. Longbow Apaches will replace attack 
helicopters currently fielded in attack helicopter units 
at battalion through corps echelons. 

Interviews were also conducted with personnel from 

OPTEC, TRAC Monterey, and TRAC White Sands Missile Range 

(WSMR), New Mexico. These interviews identified the level of 

M&S integration for the OT&E programs examined. 

2. OT&E Problem and Issue Categories 

The problems and issues are divided into five 

categories. The five OT&E problem and issue categories 

include Test Design Validation, Resource Constraints, System 

Component Stress, Safety and Environmental Concerns, and Data 

Reliability and Validity. Categories were selected when at 

least two of the six OT&E programs experienced similar 

problems and issues. The categories and general types of 

problems and issues assigned to each are described in the 

following paragraphs. 
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a. OT&E Test Design Validation Category 

Test issues and problems were categorized under 
OT&E Test Design Validation when shortcomings were associated 
wrth test design errors or omissions. The validation process 
is designed to ensure that the baseline scenarios and test 
execution plans satisfy the user and evaluator requirements 
for data collection, while simultaneously meeting the mandates 
and guidance for the conduct of OT&Es. 

A desired method for test design validation is the 
use of a pilot test. A pilot test is a rehearsal of test 
events using test participants, fully instrumented candidate 
systems, data collection resources, and test ranges and 
maneuver areas, but excluding data collected from the test 
report. A pilot test is useful in determining if the data 
collected address the criterion under consideration for 
specific events. It usually does not determine whether or not 
the test design fully addresses all critical issues 
identified. The pilot test advantages include validation of 
test scenarios, data collection plans, test facilities, and 
pre-test training with no test report impact. Disadvantages 
include time and funding requirements, and early user 
familiarization with the candidate system, test facilities, 
and test trial scenarios. 
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b. Resource Constraints Category 

Shortcomings associated with time, personnel, 
equipment, and funding availability were categorized under 
Rfesource Constraints. These shortcomings impact the OT&E test 
design and scope in terms of realism, environmental diversity, 
scale of threat and friendly force structures, and number of 
test event iterations completed. The degree to which 
resources are constrained determines the operational scope 
able to be recreated in the live test, and the type and amount 
of data available for evaluation. 

c. System Component Stress Category 

Stress is a critical element of the operational 
environment and stress level reductions on personnel or 
equipment during OT&E testing have a negative impact on the 
evaluation. Issues resulting from unrealistic equipment and 
personnel workloads, threat capabilities encountered, and 
familiarity with equipment and facilities were categorized 
under System Component Stress. OT&E test induced stress 
levels that are not representative of the operational 
environment increase the risk that evaluations are based on 
faulty or unrealistic data. 

d. Safety and Environmental Concerns Category 

Shortcomings resulting from test events that were 

eliminated or unrealistically controlled because of potential 
hazards to personnel or release of pollutants into the 
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environment were categorized under Safety and Environmental 
Concerns. As the ranges, lethality, and destructive ability 
of associated systems all increase with new technologies, 
safety concerns often limit the use of the actual system to 
very controlled circumstances not representative of a combat 
environment. Systems that create radioactivity, ozone 
depleting elements, and other environmentally hazardous by- 
products are also limited to controlled scenarios. 

e. Data Validity and Reliability Category 

Issues associated with test data interpretation or 
unusual test participant qualifications were categorized under 
Data Validity and Reliability. Live test events contain an 
infinite number of variables, providing an opportunity for 
assigning cause to undesirable outcomes and eliminating 
outliers or biasing data. The degree to which test 
participants represent typical operational users impacts data 
validity and reliability. 

C. ANALYSIS OF M&S APPLICATION TO OT&E 

OT&E problems and issues within each of the identified 
categories are detailed and the capabilities of M&S 
applications to resolve them are analyzed in the following 
paragraphs. The analysis is organized by problem and issue 
category. 
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1. TEST DESIGN 



The OT&E test design establishes the amount of data 
and the circumstances in which data are collected. The test 
design also sets the foundation for the amount of operational 
realism achieved. Creating realistic, combat conditions in 
OT&E tests have been hampered by many factors. Resource 
constraints, discussed in the following section, are a main 
cause for reduced realism. Accurate TEMP critical issue 
interpretation and understanding of the environments, 
scenarios, and forces required to replicate a representative 
operational environment have also contributed to lack of test 
realism. The M1A2, Air-to-Air Target Designator, Javelin, and 
Longbow Apache test designs had significant shortfalls. 
a. M1A2 Main Battle Tank 

Interviews with TEC personnel indicated that the 
M1A2 test was limited to an operational microcosm that 
significantly increased assessment extrapolation risks. 
Operational deployment of the M1A2 tank at the brigade or 
divisional levels could not be assessed due to the resource 
constraints which limited live testing to one battalion sized 
task force. Battalions deploy, fight, and are supported 
differently than brigades or divisions. 

A constructive simulation, such as Janus, appears 
to have application in addressing operational test design 
shortfalls similar to those experienced by the M1A2. Because 
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of the wide range of force structure, environments, threat 
forces, and combat intensities that a constructive simulation 
is capable of addressing, it is useful for assessing 
operational environments and scenarios beyond the scope of a 
resource constrained live test. 

TRAC White Sands Missile Range (WSMR) used a 
similar system to assist in the M1A2 OT&E test scenario 
development. Experiments were designed on a Janus based 
constructive simulation and experiments were iteratively 
conducted for predictive analysis. Actual test equipment were 
instrumented and data were collected on live test trials. 
Live test iterations were replicated on the simulations and 
then correlated. The correlated simulations were then used in 
other environments and scenarios within the M1A2 operational 
spectrum and analyzed. The insights gained from these 
simulations were used to assist in the M1A2 OT&E test 
conducted at TEC. OT&E test events were selected based on the 
simulated event outcomes that indicated a high probability for 
obtaining required data to address critical issues. 
(Pay an, 1994 ) 

b. Javelin Missile 

TEC interviews revealed that battlefield operating 
systems normally interoperating with Javelin missile systems 
were not included in the Javelin test plan. Javelin is 
deployed in conjunction with other anti-armor systems 
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including tanks, artillery, and close air support. This 
environment was too expensive and impractical to replicate in 
a live environment, and was omitted from the test design. 

M&S can provide a method for addressing 
interoperability issues such as those experienced by the 
Javelin OT&E. Constructive simulations are capable of 
representing other battlefield operating systems and 
integrating a candidate system's capabilities so that 
interoperability issues can be assessed. Virtual simulations 
provide methods for assessing interoperability issues with 
personnel operating associated battlefield systems within 
doctrinal guidelines. 

Some M&S efforts assisted in the Javelin OT&E test 
design. A MTM pre-test model conducted on the Javelin program 
successfully correlated live OT&E test data with models 
generated on Janus. Three scenarios with six missions each 
were conducted on the model including offensive and defensive 
Javelin employments. These resulting data provided Javelin 
employment assessments in an operational environment and OT&E 
test scenarios were changed to incorporate the MTM 
recommendations. Integrating MTM efforts into future OT&E 
test designs can improve test scenarios and environments, and 
therefore, enhance operational assessments. (IR MTM, 1993) 
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c. Air-to-Air Target Designator 

Interviews with TEC personnel indicated the Air-to- 
Air Target Designator OT&E test design validation process was 
not totally effective. The test design validation consisted 
of a paperwork review of the user and evaluator data 
requirements matched against the test design plan. A pilot 
test was not conducted and the best judgment of experienced, 
professional OT&E personnel was used to validate the test 
design plan. A test matrix was constructed, like the 
simplified illustration in Table 1. 



TABLE 1 OPERATIONAL TEST MATRIX EXAMPLE 



Test 
Events : 

Issues : 


Daylight Live 
Fire 


Night Live 
Fire 


96 Hour 
Field 
Training 
Exercise 


Can the 

system be 

effectively 

employed 

with night 

vision 

equipment? 




X 


X 


Can the 
system be 
operated in 
degraded 
modes? 


X 


X 


X 


Is the 
system 
supportable 
by user 
personnel? 


X 


X 


X 
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Operational test events that were used to address correlating 
critical issues are designated with an "X". The matrix was 
useful in ensuring that all data collection requirements were 
addressed in one or more test design events. However, this 
paperwork review was not efficient at validating the scenarios 
and environments required to fully address the issues. 

The Air-to-Air Target Designator test design plan 
validated in this manner had serious shortcomings that were 
solved during the test execution. Night operations were 
specified for the Air-to-Air designator OT&E. Because night 
operations were omitted from the design, a major test event 
rescheduling was required to accommodate the night 
environments specified. (TEXC Int,1994) 

An OT&E test design simulated on a constructive 
simulation appears to provide a method for validating the test 
design plan. The critical issues could be input into the 
simulation and simulated test trials conducted. Scenarios 
could then be adjusted to accommodate all critical issues 
before live test events begin, eliminating shortfalls such as 
those experienced during the Air-to-Air Target Designator OT&E 
test . 

M&S was used in the Air-to-Air Target Designator 
OT&E test design. The program used constructive simulations 
to design the two-on-two test scenarios. However, no 
simulation generated data were used for the report. M&S was 
used to assist in operational test scenario development and 
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test scenarios resulting from the simulations were 
successfully implemented. 

d. AH-64D Longbow Apache 

TEC interviews revealed that the Longbow Apache 
OT&E test design had threat realism shortfalls. The test 
design did not provide for many of the threat assets expected 
in an operational environment. Planned threat air defense 
were under-represented and lacked realistic electronic jamming 
capabilities. Threat air assets were under-represented and 
excluded from many planned test scenarios. 

M&S could provide methods for addressing threat 
realism shortfalls in OT&E test designs. Constructive 
simulations can replicate the full range of threat 
capabilities expected in an operational environment. 
Additionally, virtual simulations, incorporating SAFOR, can 
provide a means to include man-in-the-loop threat forces 
employed by human beings using doctrinal ly correct threat 
tactics . 

M&S was used in the Longbow Apache OT&E test design 
process. TEXCOM validated the scenarios for the test plan 
using Janus constructive simulations. In addition, the 
Longbow Hellfire missile was examined in various electronic 
warfare counter measure situations using a hardware-in-the- 
loop simulation. These pre-test simulations were used to 
establish Hellfire Pg and Pg probabilities for casualty 
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assessments. The success of the M&S assisted Longbow Apache 
operational testing indicates that M&S integration can improve 
the test design process. 

e. Test Design Summary 

In virtually all OT&Es, live testing represents 
only a small operational microcosm for the candidate system 
and data collected are therefore, limited to that microcosm. 
Live operational test evaluations are limited to these data 
collected. Operational assessments beyond the test scope must 
be extrapolated to address the system's combat effectiveness 
and suitability. As the scope of live testing is reduced, the 
evaluation validity to the operational environments not 
addressed remains unknown and risk increases. 

M&S integration into the OT&E test design appears 
to provide an opportunity for addressing areas that are beyond 
the scope of the planned testing. M&S applications, including 
MTM, constructive simulations, and virtual simulations such as 
the Hellfire hardware-in-the-loop, can enhance the OT&E test 
design process. Candidate systems in a constructive or 
integrated simulation, are able to be portrayed at every 
echelon in the Army structure, under varying physical and 
threat environments, and interoperating with other battlefield 
and logistics systems. 
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2 . RESOURCE CONSTRAINTS 



Declining DoD budgets increase the pressure to limit 
the scope of testing and the realism that can be included in 
a live test. Resource Constraints had a negative impact on 
the M1A2, Air-to-Air Target Designator, Multi-Spectral Combat 
Decoy, and Javelin tests. 

a. M1A2 Main Battle Tank, Air-to-Air Target 

Designator, and Javelin Missile 

TEC interviews indicated that these systems were 
tested at the small unit level, so the interoperability and 
impact on higher echelon units could not be ascertained by the 
OT&E tests conducted. Similarly, resource constraints limited 
the inclusion of associated battlefield operating systems and 
support elements that would normally be deployed in a combat 
environment. Interoperability issues with those systems 
cannot be addressed from test data collected. For example, 
recovering and evacuating an M1A2 tank from the battlefield to 
the corps level general support maintenance unit requires 
significant recovery and transportation assets as well as 
multi-echelon coordination. M1A2 OT&E test resource 
constraints did not allow this scenario to be addressed beyond 
the participating battalion's organic recovery capabilities. 
(TEXC Int.1994) 

Interoperability with other systems beyond the 
scope of live testing can be partially assessed through M&S. 
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For example, the TRAC WSMR simulations addressed M1A2 
capabilities in various environments and scenarios beyond the 
scope of the live trials. Using similar applications, the 
M1A2 could be portrayed interoperating with close air support, 
artillery and any other battlefield operating system including 
critical logistics support activities. This capability is 
clearly outside of the live test environment's resource 
constrained limits. 

M&S appears to have application in addressing OT&E 
test shortfalls created by resource constraints. The marginal 
cost of M&S generated operational events is extremely low as 
compared to those same events conducted in the live 
environment. Resource constraints that have a considerable 
impact on the planning of live events, become negligible in 
the M&S environment. 

An example of the wide disparity between live and 
M&S resource requirements involves Ml main battle tank 
prototypes. In 1984, evaluations of possible improvements on 
the Ml Abrams tank were carried out by using real tanks in a 
live environment. The effort took 24 months and cost $40 
million. A later effort in 1986 used a modified aircraft dome 
simulator, took only six months and cost $1 million. In 1992, 
using DIS, four variations of the Ml Abrams were operated 
against potential threats, taking only three months and 
costing $640,000. The use of M&S can reduce resource 
constraint impacts by providing more cost effective methods 
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for assessing candidate systems' operational characteristics. 
(Berry, 1992) 

b. Multi-Spectral Combat Decoy 

Interviews with TEC OT&E test officers revealed 
that resource constraints limited the scale of threat forces 
planned to be portrayed during Multi-Spectral Combat Decoy 
test events. The system was not subjected to the full range 
of threat detection capabilities that would be expected in an 
operational environment. Threat observation systems were 
limited to visual, thermal, and radar systems, excluding the 
extended range of electro-optical systems available to 
potential threat forces. (TEXC Int,1994) 

M&S appears to provide a method of addressing 
threat assets beyond the scope of live testing. Using the 
expected detection parameters of the combat decoys, test 
trials in a virtual environment provides data for threat 
capabilities that cannot be represented in a live environment. 
However, the differences in observation cross sections between 
actual and decoy systems must be quantifiable and verifiable. 

c. Resource Constraint Summary 

Resource limitations impacted realism and reduced 
the scope of the live operational test performed. Reduced 
realism resulted in evaluations that were based on data 
collected from test trials that were not strictly 
representative of the operational system, fielded in typical 
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Army units, in a combat environment. Reduced scope resulted 
in less quantitative and more qualitative evaluations on the 
candidate systems . 

• Virtual and constructive simulations can enhance 
OT&E tests that have been negatively impacted by resource 
constraints. Virtual simulations can provide methods for 
representing threat and friendly force capabilities that are 
too expensive or impractical to replicate in a live 
operational test environment. Constructive simulations can 
provide a means to assess portions of the operational spectrum 
outside of the scope of the live test. 

3. SYSTEM COMPONENT STRESS 

The stress imposed on personnel and equipment by an 
operational combat environment cannot be replicated through 
constructive, virtual, or live simulation. Therefore, live 
testing, while replicating the combat environment, is the most 
desirable, but most expensive substitute. TEC interviews 
revealed that personnel and equipment components were not 
stressed to levels expected in an operational environment in 
the Air-to-Air Target Designator, Longbow Apache, and MELIOS 
OT&Es. 

a. Air-to-Air Target Designator and AH-64D Longbow 
Apache 

The Air-to-Air Target Designator test was planned 
for two-on-two engagements, but the operational capability of 
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the designator specified force ratios of up to two-on-six 
engagements. Helicopter crews in both tests had limited duty 
cycles, not representative of an operational combat 
environment. In addition, the same ranges and maneuver spaces 
used for operational test participant training were used for 
test trials. Feuniliarity with test facilities were considered 
to improve reactions, decisions, and weapons accuracy. As 
previously discussed, threat capabilities were under- 
represented which also reduced personnel stress in these 
tests. (TEXC Int,1994) 

M&S provides methods for enhancing personnel stress 
without creating dangerous environments. Virtual simulations, 
integrating operational pilots on flight simulators with SAFOR 
simulated threat aircraft, would provide the stress associated 
with multiple aircraft engagements. Missions could also be 
extended to combat durations using simulations without 
jeopardizing crews or equipment. Data resulting would reflect 
more realistic personnel stress levels than are possible in a 
peace time environment. 

Training test participants via virtual simulations 
preserves personnel stresses induced by unfamiliar 
environments. Training could also be accomplished through 
distributed simulations using actual equipment. 

Threat induced personnel stress is enhanced through 
M&S integration. M&S applications provide the full range of 
threat capabilities. Simulations like BDS-D and SAFOR, have 
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the requisite threat signatures and capabilities. Test 
participants operating in virtual or integrated simulations 
can be opposed by the full threat capacity, controlled by 
thinking human beings in realistic environments. These threat 
forces are often more realistically reproduced in M&S than is 
practical in a live environment. Integrating virtual 
simulations enhances the stress on test participants to more 
realistic levels, therefore, operational assessments are more 
accurate. 

b. HELIOS 

HELIOS experienced limitations in the degree to 
which personnel were realistically stressed. Duty cycles for 
the MELIOS test trials were considerably shorter than the 
specified operational cycle. System induced fatigue effects 
on operators was not determined. 

Conducting trials with operational users through 
virtual simulations provides a means to extend duty cycles 
when similar live testing cannot be accommodated. The impact 
of more representative fatigue can be assessed from M&S 
generated data. The risks to personnel and equipment from 
fatigued operators is minimized by conducting extended duty 
cycles on virtual simulations rather than on actual equipment. 

c. System Component Stress Summary 

The stress imposed on personnel and equipment by an 
operational combat environment cannot be replicated through 
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constructive, virtual, or live simulation. Live simulation of 
the combat environment is the most desirable and most 
expensive means of conducting OT&E. However, the live test 
addresses only a small microcosm of the operational 
environment spectrum. 

M&S integration appears to provide a means to 
preserve some personnel stress in the conduct of actual live 
test trials. Virtual simulations using SAFOR can provide 
representative threat capabilities which could contribute to 
threat induced personnel stress. Training test participants 
through virtual and distributed simulations could also 
preserve some personnel stress associated with unfamiliar 
environments . 

4. SAFETY AND ENVIRONMENTAL CONCERNS 

Managing the safety and environmental impacts of new 
technology testing, negatively impacts on the successful 
creation of a realistic combat environment where safety and 
environmental side effects are not so important. Safety and 
environmental issues affected the M1A2, Air-to-Air Target 
Designator, Javelin, and Longbow Apache OT&E tests. 

a. Air-to-Air Target Designator and AH-64D Longbow 

Apache 

TEC test officers revealed that peace time crew 
rest requirements for helicopters in the Air-to-Air Target 
Designator and Longbow Apache tests were strictly adhered to. 
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These crew rest requirements are not representative of combat 
crew duty cycles, which are not constrained to peace time 
training restrictions. As previously mentioned, safety 
concerns associated with maneuvering more than four aircraft 
in close airspace limited Air-to-Air Target Designator trials 
to two-on-two engagements. 

M&S applications providing safe environments could 
be used to assess the impact of extended operations on 
aircraft crews. Safety concerns make this the only practical 
method for obtaining this type of operational data. 

With no actual threat to the safety of soldiers, 
safety considerations are nearly non-existent in M&S 
applications. For example, relatively close air-to-air combat 
involving more than the safety constrained two-on-two force 
ratios are probable in an operational environment. Virtual or 
constructive simulations provide a means to address issues 
involving more than two-on-two force ratios. While realism is 
reduced, safety and environmental concerns are accommodated 
through M&S integration without the total loss of data. 

b. M1A2 Main Battle Tank, Javelin Missile, and AH-64D 

Longbow Apache 

TEC interviews indicated that live fire ranges 
limited the operational employment planning for the M1A2, 
Javelin, and Longbow Apache systems. The ranges used were 
well-marked for firing limits and oriented the crews toward 
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target positions. Lasers, in all but the Javelin Missile, 
were restricted to controlled maneuver and air spaces. These 
controlled maneuver and air spaces did not provide 
representative maneuver space or engagement ranges . 
Environmental concerns eliminated operational firing of 
depleted uranium M1A2 main gun rounds due to radiation 
hazards. (TEXC Int,1994) 

Both constructive and virtual simulation 
applications provide environments that are not affected by 
these types of safety and environmental considerations. As a 
result, data generated on M&S are not biased by those factors, 
c. Safety and Environmental Concerns Summary 

Safety and environmental concerns negatively impact 
on the ability to assess the operational effectiveness and 
suitability of candidate systems. Integrating M&S into the 
OT&E process provides a means for assessing issues in an 
environment that is not constrained by safety and 
environmental concerns. Virtual and constructive simulation 
applications can be used to assess issues that cannot be 
safely or environmentally conducted in a live operational 
test. When test events are limited in number due to safety or 
environmental concerns, post-test MTM can provide a method to 
extend data and enhance operational evaluations. 
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5. TEST DATA VALIDITY AND RELIABILITY 



The myriad of variables present in a live test 
environment impact on data validity and reliability. The 
MTA2, Air-to-Air Target Designator, Multi-Spectral Combat 
Decoy, and MELIOS tests experienced data validity and 
reliability problems. 

a. Multi-Spectral Combat Decoy and MELIOS 

TEC interviews indicated that both the Multi- 
Spectral Combat Decoy and the MELIOS OT&E tests had some 
unfavorable test event outcomes that were challenged by 
parties with vested interests in the OT&E test results. 
Specific events were subsequently omitted or obscured so that 
test trials reflected more favorable results than had actually 
occurred. For example, one decoy was identified (as a decoy) 
by the threat participants because of its location in the 
maneuver area. A protest was made based on the threat 
participant's knowledge of the maneuver area and the supposed 
errant placement of the decoy. The test trial was eliminated 
from the test report. A MELIOS malfunction was traced to an 
operator error. The finding was challenged because operator 
training was conducted which should have prevented the 
malfunction. Those data were not used in the report. 

The uniform quality of M&S generated data appears 
to enhance the reliability and validity of OT&E data. M&S 
generated data are less susceptible to interpretation as the 
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M&S application controls all variables by design. Fully 
verified, validated and accredited M&S applications 
specifically designed for data collection produce uniform 
quality data which should enhance the evaluation process. 

b. M1A2 Main Battle Tank and Air-to-Air Target 

Designator 

OT&E personnel at TEC indicated that the M1A2 and 
Air-to-Air Target Designator tests had problems associated 
with atypical test participants. The OT&E test participants 
were tasked from tactical units for the test duration. As 
they were "representatives" of the tactical unit tasked with 
support, many were higher quality in terms of training, 
aptitude, experience, and attitude than typical operational 
individuals. Conducting the tests with these "golden crews" 
created some degree of test data bias. For example, the 
pilots in the Air-to-Air Target Designator test were all 
senior chief warrant officers (CW3 or CW4 ) with thousands of 
flying hours logged. Their pilot skills were considered to be 
atypically high when compared to less experienced pilots 
prevalent in the operational environment. 

M1A2 test participants were also considered to be 
atypically qualified. Training and familiarization of the 
M1A2 test crews were conducted on the same ranges and maneuver 
areas as the OT&E test trials. Decisions, response times, and 
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accuracy of weapons firing were considered superior to what 
could be expected in a combat environment. (TEXC Int,1994) 
The distributive nature of DIS provides an 
opportunity to address the "golden crew" problem. Test 
participants selected over a wide geographic area and linked 
together through DIS, can participate without leaving their 
home station. The tendency to select atypical users at home 
station would be reduced because participants remain anonymous 
and are not representing the unit as individuals. The same 
techniques could be used to train test participants without 
familiarizing them with the limited test ranges and maneuver 
areas used for actual test trials. Data collected in this 
manner would have enhanced validity and reliability, thus 
improving the evaluation. 

c. Data Validity and Reliability Summary 

Data validity and reliability are affected by the 
OT&E test scope, realism achieved, and the degree to which 
test participants represent typical users. Operational 
assessments of combat effectiveness and suitability are 
directly affected by data validity and reliability. 
Integrating M&S into the OT&E process through the use of MTM, 
virtual, constructive, and distributed simulations, enhances 
data reliability and validity, and therefore, assessments 
become more accurate. 
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D. SUMMARY 



Integration of advanced M&S techniques, technologies, and 
applications including Model-Test-Model, Janus, BDS-D, SAFOR, 
and DIS, appears to provide a means of addressing operational 
issues beyond the scope of resource constrained live testing. 
M&S appears to provide a vehicle for addressing critical 
issues that are impossible or impractical to represent in a 
live environment. Operational realism and system stress 
appear to be enhanced through M&S application. 

Integrating M&S does not replace or eliminate live 
operational testing, but rather augments and focuses live 
testing where results would provide the best data for the 
operational evaluation. The adage "you don't use M&S to 
identify what is exactly right, you use it to eliminate what 
is exactly wrong" applies to OT&E planning. Integrating M&S 
into OT&E planning reduces variability by examining critical 
issues across the operational spectrum replicated on a 
simulation . 
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V. CONCLUSIONS AND RECOMMENDATIONS 



A.* INTRODUCTION 

This thesis focused on opportunities to integrate M&S 
concepts and applications into the Army OT&E process. 
Recurring OT&E problems and issues were examined and possible 
M&S based enhancements and solutions were identified. Army 
OT&E process improvements through M&S integration were 
analyzed . 

The OT&E process has a major impact on candidate systems' 
critical acquisition decisions. From early operational 
assessments through the formal operational evaluation, the 
insights to the systems' operational characteristics are based 
on OT&Es . The accuracy of these OT&Es in predicting how 
systems will operate in a combat environment directly impacts 
on future mission accomplishment, and equally important, the 
survival of the combat troops who depend upon those systems. 

The importance of OT&Es in the acquisition process is 
underscored by the Congressional interest in the OT&E process 
and results. OT&E assessments are reported directly to 
Congress from independent OT&E sources through the DOTE. 
Continued program authorization and funding decisions are 
influenced by the operational information provided. 
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M&S integration into the OT&E process has traditionally 
been limited. In the past, M&S maturity was limited to 
physical models and mathematical simulations. Advancements in 
supporting areas from software to communications networks have 
resulted in explosive M&S growth and maturity. However, the 
OT&E community has been slow to integrate this advanced M&S 
due to the desire to protect their independent status and the 
perceived negative connotation associated with anything 
"simulated" in a testing environment. 

B. GENERAL CONCLUSIONS 

Operational combat effects on personnel and equipment 
cannot be wholly replicated through constructive simulation, 
virtual simulation, or live operational testing. Live testing 
of the combat environment provides the closest possible 
substitute. It is the most desirable, but most expensive 
method for addressing OT&E critical issues. M&S cannot, and 
should not be used in place of necessary OT&E live testing. 
However, M&S applications offer methods of augmenting and 
enhancing the OT&E process that merit consideration. 

The requirement to create holistic, representative 
operational environments conflicts with the realities of 
reduced current and projected defense funding authorizations. 
The tremendous costs associated with conducting realistic, 
comprehensive OT&Es makes them susceptible to reduced funding 
pressures. Army budgets are projected to continue the present 
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downward trend, which will not improve the prospects for 
expanding OT&E live testing. 

Interviews and other investigations indicate that there 
ate significant problems and issues associated with the Army 
OT&E process. These recurring problems and issues negatively 
impact on OT&E abilities to adequately assess the combat 
effectiveness and suitability of candidate systems, as 
required by law. Because of the identified problems and 
issues, OT&E tends to yield more favorable assessments than 
are likely to be found when the systems are employed in 
combat. This can lead to the funding of weapon systems whose 
operational effectiveness and suitability have not been 
demonstrated. 

The integration of M&S including MTM, constructive 
simulations, SAFOR, virtual simulations, and distributed 
simulations can provide some solutions to recurring OT&E 
problems and issues. M&S cannot fully represent the personnel 
stresses associated with combat environments. M&S cannot 
replicate the infinite number of variables present in live 
environments. However, integrating M&S into the OT&E process 
can provide valuable insights into the operational environment 
beyond the scope of a resource constrained, live operational 
test . 
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C. SPECIFIC CONCLUSIONS 



Specific Conclusions regarding M&S application in 
addressing OT&E recurring problems and issues are presented in 
the following paragraphs. 

1. Test Design 

a. Current Problems and Issues 

Interviews with OT&E professionals at TEC indicated 
the following OT&E test design recurring problems and issues: 



• It is generally impossible or impractical to portray the 
full range of force structures, interoperating systems, 
and potential threat capabilities faced by even the most 
basic system. 

• The current methods for planning the OT&E test, and 
validating that plan are not efficient or effective. 

b. M&S Solutions 

Integrating M&S into the OT&E test design process 
can assist planners in addressing the problems and issues, 
specifically: 

• Constructive simulations can provide a means of examining 
the candidate system's entire operational spectrum. 
Force structures, interoperating systems, and threat 
capabilities can be represented in the constructive 
environment . 

• Integrating planned OT&E live test events with 
constructive simulations and Model-Test-Model applications 
can identify scenarios and environments required to 
address critical issues. 
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2. Resource Constraints 



a. Current Problems And Issues 

TEC interviews revealed the following recurring 
problems and issues associated with resource constraints: 

• Resource constraints limit the scope of OT&E live testing. 

• The operational microcosm considered shrinks as resources 
are constrained. This increases the risk that the OT&E 
will neither accurately nor adequately address the combat 
operational environment that potential weapon systems 
might experience. 

• Resource constraints limit the degree to which realism is 
recreated in a live environment. 

b. M&S Solutions 

Integrating M&S into OT&E testing can reduce the 
impact created by resource constraints: 

• Constructive and virtual simulations can provide methods 
for OT&E to address operational issues beyond the scope of 
resource constrained live testing. 

• M&S applications are more cost effective than live test 
events and therefore, are less susceptible to the negative 
effects resulting from resource constraints. 

• Virtual simulations accommodate actual users in an 
unconstrained environment. This provides critical user 
responses in environments with realistic threat 
capabilities, interoperability with other battlefield 
operating systems, and force structures impossible to 
replicate in a live environment. 

3. System Component Stress 

a. Current Problems and Issues 

OT&E test personnel at TEC identified the following 
problems and issues related to system component stress: 
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• Stress on OT&E test participants and equipment is usually 
not commensurate with levels expected in an operational 
environment . 

•Resource constraints that limit threat capabilities, 
workload, and required interoperability functions, may 
reduce stress to unrealistic levels in the live 
environment. 

b. Mas Solutions 

M&S integration can preserve or restore more 
realistic personnel stress levels to OT&E test participants, 
specifically : 

•virtual and distributed simulations, used as a training 
vehicle for test participants, can preserve the stresses 
associated with unfamiliar weapons engagement ranges and 
maneuver areas used exclusively for live test trials. 

• Threat forces represented on virtual and constructive 
simulations, can restore the stress of confronting test 
participants with the full range and capability of 
representative threat forces, operated by threat expert 
personnel . 

• Stimulators can provide equipment stress for software 
intensive and communications systems. Other forms of 
equipment related physical stress cannot be reproduced 
through M&S applications. 

4. Safety and Environmental 

a. Current Problems and Issues 

TEC interviews identified the following safety and 

environmental issues that increasingly limit the ability to 

fully consider OT&E critical issues in a live environment: 

•New technology advancements including lasers, depleted 
uranium projectiles, and other potentially hazardous 
systems cannot be addressed in an unrestricted, live 
operational test. 



71 



• Systems that produce environmentally hazardous by-products 
are restricted to very controlled scenarios, not 
representative of a combat environment. 

• Peace time safety requirements that are significantly more 
restrictive than those expected in a combat environment, 
negatively impact OT&E assessments. 

b. M&S Solutions 

Integrating M&S into the OT&E process can provide 
solutions for data collection when scenarios are restricted by 
safety or environmental concerns: 

• M&S can provide a means to address critical issues that 
are too hazardous or environmentally polluting to conduct 
in a live test. In extreme situations, M&S may provide 
the only OT&E data obtainable for OT&E consideration. 

5. Data Validity and Reliability 

a. Current Problems and Issues 

Interviews with TEC OT&E personnel revealed the 

following data validity and reliability problems and issues: 

• Data collected from the OT&E microcosm must be 
extrapolated to the operational spectrum not addressed 
within the live OT&E scope. As OT&E test scope becomes 
smaller and more constrained, data validity and 
reliability risks increase. 

• The myriad of variables present in a live OT&E environment 
invites questionable interpretation of the results. False 
causes may be assigned to some undesirable events that 
eliminate, obscure, or reduce the severity of the failure. 

• OT&E test participants are often not representative of the 
typical operational user. Data validity and reliability 
suffer when "golden crews" (personnel who are atypically 
high in aptitude and attitude) are used to represent the 
"typical" user. 

• Training test participants on the same ranges and maneuver 
areas used for OT&E test trials creates unrealistic 
responses. This familiarity improves response times, 
anticipation, and weapons accuracy. 
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b. M&S Solutions 



Integrating M&S into the OT&E process can assist in 
addressing data validity and reliability related problems and 
issues : 

• M&S applications can provide a means to augment data 
required for quantitative evaluation. 

• Data generated on M&S applications have controlled 
variables, virtually eliminating the opportunity for event 
interpretation and manipulation. 

• DIS can provide an effective means for addressing the 
"golden crew" problem. Data collection and test 
participant training via DIS preserve the actions, 
reactions, and decisions of representative operational 
users . 



D. RECOMMENDATIONS 

Implementing the following recommendations should enable 
M&S to enhance and improve the Army OT&E process. 

1. The Army should continue and expand existing M&S 
efforts in requirements generation, data augmentation, and 
OT&E. 



Cooperative M&S efforts between OPTEC and TRADOC, such 
as the M1A2 and Javelin programs, will enhance the OT&E 
process through early and accurate critical issue 
identification and test design validation. OT&E test data 
will be improved and combat effectiveness and suitability 
assessments will be more accurate. 
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2. M&S applications that have the fidelity to integrate 
candidate system capabilities and parameters for OT&E should 
be developed. 

The M&S applications developed should portray existing 
threat and friendly force structures in varying environments 
through theater level. These applications should include 
constructive and virtual simulations capable of examining the 
impacts on threat capability, battlefield interoperability, 
and logistics supportability issues. This will provide 
methods for examining the operational spectrum beyond the live 
test scope. Extrapolation of live test data via M&S will 
enhance data and evaluation accuracy. 

3. Verification, Validation and Accreditation (W&A) 
requirements for previously validated applications should be 
limited to the candidate systems' capabilities emd parameters 
added . 

The W&A process for M&S applications is time 
consuming and expensive. M&S applications with standing W&A 
approval could enhance M&S integration into the OT&E process. 

4. Equipment simulators at user locations should be 
linked through DIS for virtual simulations integrating a wide 
variety of typical users for operational assessments. 

Test participant training at home station through DIS 
could improve OT&E data validity during live test trials. 
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E . FURTHER RESEARCH 



Further research is recoimnended on the following subjects 
relating to this thesis: 

1 . Research should be conducted to determine the 
differences between test participant actions, reactions, and 
decisions in live versus simulated environments. 

The results of this research should help establish the 
degree to which M&S generated reaction and decision data 
correlate with live operational testing. 

2 . Research should be conducted on the impacts of the 
future digitized battlefield on live and simulated OT&E test 
events . 

The effects of the future digitized battlefield on 
tactics and user decisions must be replicated in OT&E and 
related M&S applications. This will be required to accurately 
portray the operational environment under battlefield 
digitization in future M&S applications. 

3. The compatibility of current and future weapon system 
training simulation devices with DIS and other M&S 
architectures should be investigated. 

Dual use of training simulation devices for training 
and OT&E testing via virtual and distributed simulations could 
enhance the M&S integration into the OT&E process. 

4. The integration of artificial intelligence with 
operationally related M&S applications should be analyzed. 
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Candidate system employment via artificial 
intelligence simulations could provide effectiveness and 
suitability assessments while varying doctrine and tactics. 
Threat forces controlled through artificial intelligence 
simulations could provide probable threat adjustments to the 
employment of candidate weapon systems. 
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APPENDIX 



Acq Acquisition 

ADATS-A Acq and Dev of Threat Simulators Activity 

AFOTEC . Air Force Operational Test and Evaluation Command 

AMC Army Materiel Command 

AR Army Regulation 

ARPA Advanced Research Projects Agency 

ASA Assistant Secretary of the Army 

ASA(RDfitA) . . . ASA (Research, Development, and Acquisition) 

ASARC Army Systems Acquisition Review Council 

ATD Advanced Technology Demonstration 

BDS-D . . Battlefield Distributed Simulation - Developmental 
BLG Battle Laboratory Guide 

C4I . Cmd, Control, Communications, Computers, Intelligence 

CEP Concept Evaluation Program 

COEA Cost and Operational Effectiveness Analysis 

DAB Defense Acquisition Board 

DIS Distributed Interactive Simulation 

DMSO Defense Modeling and Simulation Office 

DoD Department of Defense 

DoDI DoD Instruction 

DOTE Director of Operational Test and Evaluation 

DSI Defense Simulation Internet 

DT&E Developmental Test and Evaluation 

DUSA Deputy Under Secretary of the Army 

DUSA(OR) DUSA (Operations Research) 

ECM Electronic Counter Measures 

EUTE Early User Test and Evaluation 

FDTE .... Force Development Testing and Experimentation 

FOTE Follow-on Operational Test and Evaluation 

FY Fiscal Year 

lOTE Initial Operational Test and Evaluation 

1ST Institute for Simulation and Training 

JT&E Joint Test and Evaluation 

LAM Louisiana Maneuvers 

LRIP Low Rate Initial Production 
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LTG 



Lieutenant General 



M&S Modeling and Simulation 

MELIOS . . . Mini Eyesafe Laser Infrared Observation System 

MS Milestone 

MTM Model-Test-Model 

OEC Operational Evaluation Command 

OPEVAL Operational Evaluation 

OPTEC Operational Test and Evaluation Command 

OR Operations Research 

ORD Operational Requirements Document 

OSD Office of the Secretary of Defense 

OT Operational Test 

OTSA Operational Threat Support Agency 

OT&E Operational Test and Evaluation 

OTEA Operational Test and Evaluation Agency 

Ph Probability of Hit 

Pk Probability of Kill 

PDU Protocol Data Unit 

R&D Research and Development 

RD&A Research, Development and Acquisition 

RDT&E Research, Development, Test & Evaluation 

S&T Science and Technology 

SAFOR Semi-Automated Forces 

SIMNET Simulation Network 

STRICOM . . Simulation, Training and Instrumentation Command 

T&E Test and Evaluation 

TEC TEXCOM Experimentation Center 

TECO Test and Evaluation Coordination Office 

TEMP Test and Evaluation Master Plan 

TEXCOM Test and Experimentation Command 

TMDE Test, Measurement, and Diagnostic Equipment 

TRAC TRADOC Analysis Command 

TRADOC Training and Doctrine Command 

USAPS United States Army Policy Statement 

use United States Code 

W&A Verification, Validation, and Accreditation 

VMS Virtual Memory System 
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